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ABSTRACT 


Sometimes  hard,  quotable  data  is  not  available  for  the  purpose  of 
reliability  analysis.  Some  of  the  reasons  for  non-availability  of  data  are: 

(a)  Field  data  is  improperly  or  not  recorded  at  all. 

(b)  Data  is  of  a proprietary  nature  and  cannot  be  released. 

(c)  Newness  of  concept  or  design  prevents  one  from  having 
a recorded  history  of  field  experience. 

(d)  Failures  occur  with  such  a low  frequency  that  many  years 
(decades)  are  required  to  obtain  significant  failure  data. 

Initiation  of  new  data  collection  programs  are  often  not  feasible,  in  terms 
of  money  as  well  as  time.  Specifically,  under  the  above  circumstances  it 
is  costly  to  set  up  a laboratory  life  test  or  carefully  monitor  equipment  al- 
ready deployed  in  the  field.  Furthermore,  in  the  safety  area  it  is  un- 
thinkable to  3et  up  laboratory  tests  that  expose  humans  to  additional 
hazards. 

As  an  aid  in  providing  initial  approximate  numerical  values  where 
no  data  exists,  the  consensus  approach  is  recommended.  The  consensus 
approach  is  really  just  an  averaging  of  intelligent  guesses  by  experts 
based  on  experience  and  partial  data.  In  fact,  sometimes  actual  proprie- 
tary data  is  available,  and  can  only  be  furnished  if  it  is  reported  as  an 
"expert  guess.  " 

In  this  paper  we  will  discuss  analytical  consenses,  meaning  a sys- 
tematic approach  involving  careful  descriptions  of  the  assumptions  about 
the  experts  and  the  resulting  choice  of  statistical  approach  used  in  anal- 
yzing their  opinions.  This  technique  is  different  from,  but  has  some  of 

* 

the  same  flavor  as  the  technique  used  for  estimation  of  prior  densities  in 
Bayesian  Analysis  and  the  Delphi  Technique  for  forecasting. 
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In  the  consensus  approach  we  ask  a panel  of  experts  to  give  their  < 

estimates  (guesses)  for  the  basic  probabilities  or  failure  rates  for  a well  , 

described  system.  The  correctness  of  the  results  depends  upon  the  knowl- 
edge and  unbiasedness  of  the  experts  and  the  size  of  the  sample  (panel). 

The  results  based  on  the  first  round  of  the  estimation  can  be  fed  back  to 

r I 

the  experts  with  the  request  that  they  provide  revised  estimates.  If  no  i 

names  are  used  in  the  distributed  results,  some  experts  who  are  unsure 
of  themselves  and  have  guessed  very  high  or  very  low  may  tend  to  revise 

their  guesses  closer  to  the  mean  of  the  group.  If  names  are  released,  1 

and  one  of  the  experts  is  well  known  and  respected  in  the  field,  his  opin- 
ions may  dominate  the  second  round.  i 

To  control  the  accuracy  of  the  end  results,  the  experts  may  be 
asked  to  rate  their  expertise  on  a suitable  scale  depending  upon  whether 
the  guess  is  backed  up  by  hard  data,  partial  data,  data  on  similar  prod- 
ucts, etc. 

The  expert  may  be  requested  to  give  a point  estimate,  a range 
estimate  (minimum  and  maximum)  or  a three -point  estimate  (minimum, 
maximum  and  mean). 

The  estimates  of  the  experts  are  then  combined  using  a choice  of 
techniques: 

(1)  Maximum  Likelihood  Extimation  (MLE) 

(2)  An  Unbiased  Linear  Combination  (ULC) 

(3)  A Linear  Unbiased  Minimum  Variance  (LUMV)  estimator 

(4)  Preanalysis  of  the  data  to  determine  if  clustering  is  present 
(e.  g. , separate  groups  of  optimists  and  pessimists)  and  seg- 
regation of  the  data  into  homogeneous  groups.  Subsequently, 
one  of  the  above  estimation  techniques  is  applied  to  each 
group  separately. 
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All  these  techniques  result  in  relatively  simple  formulae  which 
are  presented  along  with  the  mathematical  development  in  Appendix  B. 

These  techniques  have  been  applied  to  the  shock  hazard  problem 
in  power  lawn  mowers.  This  study  was  conducted  in  conjunction  with  the 
Consumer  Product  Safety  Commission's  (CPSC)  Safety  Standard  Study 

I 

Committee  chaired  by  Consumers  Union.  This  Committee  participated 
in  the  formulation  of  safety  standards  presently  being  considered  by  the 
CPSC.  The  results  of  an  analysis  of  the  relative  safety  against  shock 
hazards  for  two  lawnmower  designs  is  included.  The  basic  event  proba- 
bilities were  generated  by  consensus  analysis  of  about  10  expert  opinions, 
and  combined  using  fault  tree  analysis  to  yield  equipment  safety. 
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1.0  Introduction 

1.  1 Objectives : The  purpose  of  this  report  is  threefold:  (a)  to  illustrate  how  the 
well-known  analytical  techniques  of  fault  tree  analysis  can  be  used  to  study  system 
safety;  (b)  to  document  a study  using  fault  tree  techniques  which  was  performed  to 
quantitatively  investigate  the  relative  electric  shock  safety  of  two  different  elec- 
tric power  lawn  mower  designs;  (c)  to  develop  new  statistical  techniques  for  han- 
dling in  a scientific  fashion  the  "guesstimates"  of  a group  of  experts  as  to  the 
necessary  elemental  probabilities  in  safety  analyses  (consensus  methods). 

1. 2 Fault  Trees:  The  principles  of  fault  tree  analysis  have  been  well  documented 

in  the  literature.  ^>2,  3,  -)  -phe  major  reason  for  constructing  a fault  tree 

is  to  provide  an  analytical  tool  (in  this  case  a topological  vehicle)  about  which  to 

% 

focus  the  safety  analysis.  The  analysis  begins  by  a single  engineer  or  a group 
listing  all  the  possible  modes  of  failure  in  the  system  and  tracing  in  a cause  and 
effect  manner  their  consequences.  Those  modes  which  can  cause  injury  or  fatality 
(and  in  some  cases  property  damage)  are  retained  for  the  safety  analysis.  Borrow- 
ing a term  from  reliability  analysis,  this  procedure  can  be  called  failure  modes 
and  effects  analysis,  FMEA.  The  fault  tree  is  then  constructed  as  a diagram  which 
has  a tree  top  event,  injury  or  fatality,  and  lower  events  which  branch  out  to  in- 
clude the  combinations  of  elementary  events  (branch  tips)  which  result  in  the  tree 
top  event.  Logical  AND  and  OR  symbols  are  used  to  connect  the  events  (faults) 
in  the  fault  tree,  which  derives  its  name  from  the  "branch  like"  structure  beneath 
the  top  event. 

1.  3 Lack  of  Data:  Safety  analysis  shares  the  problem  with  reliability  analysis 
that  there  is  almost  always  a problem  in  obtaining  sufficient  data.  The  fundamental 
reason  for  the  problem  is  that  we  are  dealing  with  rare  events  (which  we  wish  to 
cause  to  be  rarer  still)  and  there  is,  therefore,  little  available  failure  data.  It  is 
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very  costly  in  terms  of  dollars  and  time  to  set  up  a laboratory  life  test  or  care- 
fully monitor  equipment  already  deployed  in  the  field  to  accumulate  such  data. 
Invariably  such  data  gathering  deals  with  previous  designs  and  must  be  extrapo- 
lated with  care  to  apply  it  to  new  designs. 

In  the  case  of  the  field  of  reliability  which  is  25-30  years  old,  much  data 

has  been  accumulated.  Although  a large  portion  of  this  data  is  questionable  with 

regard  to  accuracy  and  applicability,  there  is  at  least  a partial  data  base  to  build 

on.  Whenever  an  elementary  event  is  included  in  the  reliability  data  base,  this 

data  can  be  used.  Since  a large  number  of  elementary  events  are  important  to 

safety  but  not  reliability  analysis,  and  safety  analysis  is  perhaps  only  5-10  years 

old,  a paucity  of  safety  data  exists.  Also  most  of  the  existing  reliability  data 

has  been  accumulated  to  support  DOD  or  NASA  projects  and  was  paid  for  by  them 

(e.  g.  , the  Minuteman  Missile  Program,  the  Apollo  Program).  The  only  large, 

funded,  safety  projects  which  have  developed  such  data  are  those  involving  nuclear 

safety.  The  work  on  atomic  weapons  is  still  deeply  classified,  and  the  Rasmussen 

Study  ^ of  the  Safety  of  Nuclear  Power  Reactors  has  only  been  published  in 

* 

the  last  year. 

In  view  of  the  fact  that  lack  of  data  was  one  of  the  major  stumbling 
blocks  in  this  analysis,  the  NRC  (successor  to  the  AEC  which  sponsored  the 
Rasmussen  Study)  is  now  sponsoring  a study  to  construct  a nuclear  power  re- 
actor failure  rate  data  manual.  Similarly,  the  difficult  and  costly  job  of  docu- 
menting "field  failures"  of  deployed  nuclear  power  reactors  is  in  its  embryonic 
stages  and  is  being  actively  pursued  by  the  NRC. 
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1.  4 The  Consensus  Approach:  As  an  aid  in  providing  initial  approximate  num- 
bers where  no  data  exists,  the  consensus  approach  is  recommended.  The  con- 
sensus approach  is  really  just  an  averaging  of  intelligent  guesses  and  partial 
data  by  experts.  In  this  report  we  will  deal  with  analytical  consensus,  meaning 
a more  careful  description  of  the  assumptions  about  the  experts  and  the  resulting 
choices  of  statistical  approaches  to  the  averaging  of  the  data.  This  work  is  dif- 
ferent from,  but  has  some  of  the  flavor  of,  the  techniques  for  estimation  of  prior 
densities  in  Bayesian  Analysis. 

The  basis  of  the  entire  approach  is  that  the  experts  to  be  queried  have 
access  to  implicit  or  explicit  data.  First  of  all,  in  some  cases  an  expert  will 
have  actual  data  which  because  of  valid  or  imagined  proprietary  reasons,  he  is 
reluctant  to  reveal.  For  example,  is  it  in  the  best  interests  of  company  X to 
reveal  that  in  the  last  ten  years  during  which  they  made  and  sold  1, 000,  000 
toasters  per  year,  three  people  were  electrocuted  by  inserting  a fork  into  that 
toaster  to  retrieve  a jammed  piece  of  toast  without  pulling  the  plug.  Certainly 
an  engineer  would  have  fewer  reservations  in  performing  his  calculations  in  pri- 
vate and  basing  his  estimate  on  the  results.  In  the  former  case  we  can  make  our 
own  computations  to  check  the  accuracy  of  his  interpretation,  and  we  know  our 
result  is  backed  up  by  hard  data.  In  the  latter  case  we  have  no  less  accurate  a 
result;  however,  unless  we  ask  our  expert  for  not  only  his  estimate,  but  some 
feeling  for  whether  or  not  it  is  based  on  data,  we  cannot  comment  on  how  believ- 
able the  estimate  is.  Let  us  suppose  that  expert  Y at  another  toaster  company 
can  find  no  data  in  his  company's  legal  files  of  any  electrocution  suits  involving 
the  300,  000  toasters  per  year  his  company  sells.  He  might  estimate  one  such 
occurrence  and  calculate  an  upper  bound  on  the  probability.  Suppose  a third 
engineer  at  company  Z which  makes  power  hand  tools  but  no  toasters  is  asked 
for  his  estimate.  Suppose  he  is  familiar  with  shock  problems  with  his  own 
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products  and  has  just  read  a study  by  the  Consumer  Product  Safety  Commission 
about  electrocution  and  appliance  cords.  He  learns  that  most  of  the  injuries  are 
due  to  young  children  contacting  alive  surface  with  their  mouth.  Other  causes 
are  much  smaller.  Suppose  he  estimates  that  the  fork-toaster  event  should  be 
about  as  probable  as  one  of  the  "other  causes"  and  bases  his  estimate  on  this 
line  of  reasoning. 

Based  on  a knowledge  of  the  three  experts’  sources  of  information,  we 
would  give  most  weight  to  expert  X,  less  to  expert  Y,  and  least  to  expert  Z. 
Lacking  any  knowledge  of  their  expertise  or  data  sources,  we  would  treat  them 
equally.  Section  4.  0 and  the  Appendix  of  this  report  formalize  some  of  these 
assumptions  and  their  statistical  consequences. 

We  now  explore  hew  to  test  a second  estimate  by  a new  expert  who  also 
works  at  company  X.  We  will  call  our  old  expert  X^  and  our  new  one  X9.  If  we 
knew  that  X^  and  X ^ were  using  the  same  data  and  obtained  the  same  results, 
very  little  new  information  has  been  added  (except  that  we  are  now  less  inclined 
to  check  X^'s  math  since  X^  has  already  done  this).  If  we  know  that  X^  worked 
from  a different  source  or  used  a different  approach  than  X^,  then  we  must  include 
him  as  a fourth  independent  estimate. 

In  attempting  to  apply  the  consensus  method,  we  may  borrow  from  the 
related  Delphi  Method^6^'^  often  used  to  estimate  future  events,  and  subject 
our  experts  to  feedback.  By  feedback  we  mean  that  a second  round  of  guesses 
is  solicited  after  the  results  of  the  first  round  have  been  distributed.  If  no  names 
are  used  in  the  distributed  results,  some  experts  who  are  unsure  of  themselves 
and  have  guessed  very  high  or  low  may  tend  to  revise  their  guesses  closer  to 
the  mean  of  the  group;  whereas  individuals  very  sure  of  themselves  would  "stand 
pat".  If  names  were  published  with  the  results,  the  senior  or  recognized  expert 
might  tend  to  shift  the  guesses  closer  to  his  estimate.  It  is  not  clear  whether 
feedback  does  result  in  improved  estimates. 


5 

When  guesses  are  given,  they  can  be  either  a single  point  or  a range. 
Although  only  a point  guess  was  used  in  the  Lawn  Mower  Study  which  is  discussed 
in  Section  5,  the  authors  recommend  an  interval  guess  (high  and  low  values) 
should  be  used.  This  is  discussed  in  detail  in  Section  4 and  in  the  Appendix. 

One  must  also  be  concerned  with  a uniform  bias  on  the  part  of  the 
experts,  perhaps  due  to  a mood  or  an  influential  report  or  study  which  many 
of  them  have  read.  The  best  example  of  this  is  the  statistician's  fable  of  the 
time  many  centuries  ago  when  a statistician  wished  to  estimate  the  height  of 
the  Emperor  of  China.  He  asked  1 million  Chinese,  and  because  of  the  large 
size  of  the  sample,  estimated  his  height  as  ten  feet  with  an  extremely  small 
variance.  Unfortunately,  none  of  these  Chinese  had  seen  the  Emperor  and 
guessed  he  must  be  a super  human  being.  In  the  sense  of  our  problem,  none 
of  them  were  experts  since  none  of  them  had  ever  seen  the  Emperor. 

The  authors  must  emphasize  that  the  merit  of  the  consensus  method 
is  based  upon  the  existence  of  some  factual  quantitative  knowledge  about  failures 
by  the  experts.  If  the  experts  know  little  more  about  quantitative  values  than, 
say,  a random  selection  of  the  general  public,  then  the  results  will  be  of  no  value. 

2.  0 High  Level  Fault  Tree  for  a Power  Lawn  Mower 

2.  1 Purpose  of  the  Study:  A fault  tree  diagram  has  many  uses.  In  this  high  level 

. . . * 
study  it  is  used  to  insnre  that  a certain  proposed  standard  does  not  omit  any 

hazards  (injury  modes).  Fault  tree  diagrams  will  also  be  used  in  Section  3.  0 to 

insure  that  the  proposed  standard  guards  against  as  many  of  the  injury  causative 

events  as  possible.  The  object  of  the  studies  in  Sections  2.  0 and  3.  0 is  to  develop 


Standard  developed  by  the  members  of  the  Consumers  Union  Study  Group 
for  use  in  the  development  of  a Consumers  Product  Safety  Commission  Safety 

Standard  for  Power  Lawn  Mowers. 
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the  expression  for  the  probability  of  hazard  and  to  study  qualitative  and  quanti- 
tative effects  of  various  sections  of  the  safety  standard  suggested  by  members 
of  the  study  group. 

2.  2 Correlation  of  Study  to  Safety  Standard:  As  previously  stated,  the  fault  tree 
analyses  of  this  section  and  the  following  section  were  performed  as  an  aid  in 
the  evolution  and  study  of  a safety  standard  for  power  lawn  mowers.  Consumers 
Union,  under  contract  to  the  U.  S.  Consumer  Product  Safety  Commission,  called 
together  a group  of  interested  consumer  and  industry  representatives  to  work  on 
the  development  of  a draft  standard.  The  written  documents,  evolving  drafts, 
testimony,  etc.  , would  be  carefully  considered  when  subsequently  the  Consumer 
Product  Safety  Commission  established  the  government  standard. 

Professor  Shooman  volunteered  his  services  and  was  chosen  as  a con- 
sumer member  of  this  study  group.  Prior  to  receiving  a copy  of  the  first  draft 
of  the  standard  prepared  by  Consumers  Unior  as  a working  document  for  the 
first  round  of  meetings,  he  prepared,  with  the  assistance  of  his  graduate  class 
in  reliability  analysis,  the  fault  tree  given  in  Fig.  Bl.  Several  days  later  when 
a draft  copy  of  the  standard  arrived,  a detailed  comparison  of  the  standard  and 
the  fault  tree  showed  that  except  for  minor  differences  in  grouping,  both  docu- 
ments dealt  by  and  large  with  the  same  hazards.  The  authors  view  this  result 
as  a check  on  the  completeness  of  the  documents. 

2.  3 Fault  Tree  Analysis:  A first  level  fault  tree  is  depicted  in  Fig.  Bl.  Various 
hazards  H^,  . . . , ^ are  linked  by  OR  gates  such  that  the  occurrence  of 

any  of  the  hazards  Hj,  . . . , leads  to  the  occurrence  of  the  top  event. 
Hazards  H^,  are  as  follows: 
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Hj  : Flying  objects 
: Noise  (Hearing) 

H ^ : Fire  (Gas  or  Electric) 

H,  : Limbs  Touch  31ade 

Hg  : Electric  Shock  (Gas  or  Electric) 

: Burns 
Hy  : Explosion 
Hg  : Recoil  Starter 

: Curious  Children 
H^:  Out  of  Control 
H^:  Flying  object. 

Basically  the  hazards  and  are  the  same,  i.  e.  , fly.ng  oojects. 

The  difference  lies  in  their  effects.  is  the  hazard  where  flying  oeiects  cause 

injury  to  the  operator,  whereas  is  the  hazard  where  flying  objects  cause  in- 

jury to  bystanders,  i.  e.  , people  other  than  the  operator. 

Z.  4 Probability  Expression:  The  probability  of  injury  Pj  is  our  numerical  risk 
index  (or  probability  of  safety,  P^  = 1 - P^)  and  is  related  to  the  hazards  by  a 
probability  expression. 

Using  the  fault  tree  diagram  and  the  principles  of  combinatorial  logic  and 
probability^ ei'  can  write  an  expression  for  the  probability  of  injury  P^.  The 
legend  on  Fig.  B2  indicates  that  the  OR  logic  symbol  is  union  (i.  e.  , one  or  more 
events  must  occur  before  the  output  event  can  occur)  and  the  AND  symbol  is  inter- 
section (i.  e.  , the  output  occurs  only  if  all  inputs  occur).  The  symbol  + is  used 
in  our  equation  for  union  (OR)  and  the  symbol  • for  intersection  (AND). 
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In  Fig.BI,  injury  occurs  if  one  or  more  of  the  11  hazards  occur;  there- 
fore, the  expression  consists  of  the  union  of  the  11  hazards; 

Pj  = P(Hj  + + H3  + . . . + Hj  p (1) 

Lower  level  fault  trees  can  be  developed  for  each  of  the  hazards  H^,  H^,  . . . , 
and,  similarly,  probability  expressions  can  be  developed  for  each  of  the  hazards 
Hj , H£,  . . . , Hj  j . (cf.  Section  3.  0) 

2.  5 Use  in  Design  and  Standard  Making:  The  fact  that  the  fault  tree  can  be  used 
from  the  conceptual  stages  through  completion  of  the  design  makes  it  particularly 
useful  in  studying  the  effectiveness  of  the  proposed  standard.  A fault  tree  for 
the  top  event  (injury)  is  drawn  and  the  probability  expression  for  the  top  event  is 
developed.  It  is  then  numerically  evaluated  using  the  probabilities  of  each  event 
(before  implementation  of  the  proposed  standard).  The  general  probability  ex- 
pression pinpoints  the  events  having  maximum  impact  on  the  probability  of  the 
top  event,  thus  helping  us  to  direct  our  efforts  toward  regulations  which  will  re- 
duce the  probability  of  such  events.  Numerical  values  of  th-e  probabilities  evalu- 
ated using  reduced  event  probabilities  (after  implementation  of  the  standard)  will 
give  us  a quantitative  basis  for  justifying  the  proposed  standard. 

3.  0 Lower  Level  Fault  Tree  for  Power  Lawn  Mower  Electric  Shock  Hazard 

3.  1 Purpose  of  Study:  As  was  previously  mentioned,  one  purpose  of  this  study 
is  to  develop  an  expression  for  the  safety  probability  of  electric  shock  hazard  and 
to  study  the  qualitative  and/or  quantitative  effects  on  safety  of  various  clauses  in 
the  proposed  standard.  Specifically,  we  wish  to  compare  the  shock  hazard  of 
grounded  type  lawn  mowers  with  the  doubly  insulated  type. 
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E^:  The  event  live  wire  touches  conductive  part  of  the  mower. 

This  could  be  due  to  improper  or  loose  connection  at  any  terminal 
on  the  lawn  mower. 

Eg:  The  event  insulation  of  lawn  mower  does  not  work. 

This  could  be  either  due  to  manufacturing  defect,  environment,  or 
prolonged  and  rough  use.  Moisture  or  chemical  deposition  may 
render  conductivity  to  insulating  surfaces  or  cause  corrosion.  Ex- 
cessive heat  or  rough  use  may  develop  cracks  in  insulation  making 
it  thu3  ineffective. 

3.  3 Fault  Tree  Analysis -Doubly  Insulated  Type:  A fault  tree  for  electric  shock 
hazard  of  a doubly  insulated  lawn  mower  is  depicted  in  Fig.  B3. 

All  the  events  (primed)  have  the  same  meaning  as  the  corresponding 
events  in  the  case  of  grounded  type  lawn  mowers  except  the  following  event: 

Eg  : The  event  protecting  insulation  does  not  work. 

This  could  be  either  due  to  a manufacturing  defect,  environment, 
or  prolonged  and  rough  use. 

For  readers  not  familiar  with  insulation  terminology  we  include  suggested 
♦ 

definitions  : 

Double  Insulation:  An  insulation  system  comprised  of  both  functional  insulation 
and  supplementary  insulation,  with  the  two  insulations  physically  separated  and 
so  arranged  that  they  are  not  simultaneously  subjected  to  the  same  deteriorating 
influences  (temperature,  contaminants,  etc.)  to  the  same  degree.  (See  Fig.  B4. ) 


These  definitions  are  taken  from  working  documents  of  the  CU/CPSC 
(Ref.  8) 

3tudy 
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3.2  Fault  Tree  Analysis  - Grounded  Type  Lawn  Mower:  A fault  tree  for  electric 
shock  hazard  of  a grounded  type  of  lawn  mower  is  depicted  in  Fig.  B2. 

The  various  possible  events  are  linked  by  standard  OR  and  AND  gates. 
The  outputs  of  lower  level  gates  are  fed  as  inputs  to  the  higher  level  gates,  thus 
leading  to  the  top  event  electric  shock  (e.  g.  , hazard  5 in  Fig.  Bl). 

A further  explanation  of  the  various  events  is  given  below: 

E^:  The  event  that  a person  is  grounded  or  has  a low  resistance  while  oper- 

ating the  lawn  mower.  (Low  resistance  is  that  resistance  which  is  suf- 
ficiently small  to  cause  an  electric  shock  for  the  specified  operating 
voltag  e. ) 

Eq:  The  event  that  the  wire-plug  of  the  mower  is  in  while  the  mower  is  being 

repaired  or  adjusted. 


E!: 


E2: 


E5: 


E6: 


The  event  that  a person  touches  a "live"  part  of  the  lawn  mower.  The 
live  part  considered  here  is  a part  which  is  normally  live  when  a mower 
is  connected  to  the  supply,  and  not  a part  that  has  become  live  due  to  a fault. 
The  event  that  a person  touches  cut  (E^)  or  damaged  (E^)  cord  when  the 
mower  is  connected  to  the  supply. 

The  event  that  the  cord  is  cut  due  to  some  sharp  object  either  while 
the  mower  is  in  operation  or  stored. 

The  event  insulation  of  the  cord  does  not  work.  (Damage  due  to  environ- 
ment, prolonged  and  or  abusive  use,  etc. ) 

The  event  that  a person  touches  the  conductive  p«  rt  of  the  mower  (includes 
handle,  blade,  etc.)  when  the  mower  is  connecter  to  the  supply. 

The  event  grounding  does  not  work. 

This  event  is  comprised  of  one  or  more  of  the  following: 

i)  consumer  does  not  have  a grounded  outlet  box, 

ii)  ground  wire  is  either  broken  or  disconnected  at  the  grounding  terminal, 

iii)  high  impedance  grounding  circuit  exists. 
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Protective  Insulation:  An  independent  insulation  provided  in  addition  to  the 
functional  insulation  to  ensure  protection  against  electric  3hock  in  case  of 
failure  of  the  functional  insulation.  An  enclosure  of  insulating  material  may 
form  a part  or  the  whole  of  the  protective  insulation. 


3.  4 Probability  Expression  - Exact:  Before  we  derive  a hazard  probability  ex- 
pression, we  will  briefly  discuss  the  expansion  of  the  probability  of  a union. 

Assuming  that  event  Y is  the  union  of  two  disjoint  events  X^+X^,  we 

obtain: 


Y = Xj  + X2 


P(Y)  = P(Xt  + X2)  = P(XX)  + P(X2) 


(2) 


If  we  consider  the  case  of  two  events  Xj  and  X2  which  are  not  disjoint, 


we  obtain: 


P(Xj  + X2)  = [PtZj)  + P(Z2)]  + [P(Z3)  + P(Z2)]  - P(Z2) 


= P(XX)  + P(X2)  - P(Xt  • x2) 


(3) 


Thus  when  events  X^  and  X2  are  not  disjoint,  we  must  subtract  the 
probability  of  the  intersection  of  X^  and  X2  from  the  sum  of  the  probabilities. 

Similarly,  we  can  show  that  if  X^,  X2>  and  X^  are  not  disjoint  then: 

p(xt  + x2  + X3)  = P(X1)  + P(X2)  + P(X3) 

- P(Xj  -x2)  - P(x2  -x3) 

- P(Xj  • X3)  + P(Xj  • Xjj  • X3)  (4) 

* 

The  general  expression  for  the  union  of  n events  follows  by  analogy. 

From  Fig.  B2  we  see  that  the  top  event  H (electric  shock)  occurs  when 

the  event  E and  at  least  one  of  the  events  E.,  E0,  and  E_  occur  simultaneously.  Hence, 
g Act- 

P(H)  = P[Eg-(EA  + Eb  + Ec)}  (5) 


]|C 

M.Shooman,  op.  cit.  , Eq.  2.12. 
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Events  E^,  E^,  Eg,  and  E^  are  independent  and  therefore: 


P(H)  = P(E  ) P(EA+EB+EC) 

= P(Eg){p(EA)+P(EB)+P(Ec)-P(EA-  Eg) 

-P(EB-Ec)-p(EA.Ec)+p(EA-EB-Ec)} 

= P(E  ){P(EA)+P(EB)+P(EC)-P(EA)P(EB) 
-P(Eb)P(Ec)-P(Ea)P(Ec)+P(Ea)P(Eb)P(Ec)}  (6) 

Similarly,  from  Fig.  B3  we  derive  an  expression  for  probability  P(H')  as: 

P(H')  = P(E  ){p(E^)+P(Ey+P(Ey-P(E^)P(Ey 

-p(Eyp(Ey-p(E^)P(Ey+p(E^)P(E^)P(Ey}  (7) 

3.  5 Approximate  Probability  Expressions:  We  know  that  the  ultimate  use  of 
the  above  expressions  is  to  estimate  the  numerical  values  of  the  probability  of 
hazard  occurrence  so  that  the  probabilities  of  hazards  for  two  different  types 
(e.  g.  , grounded  type  and  doubly  insulated  type)  can  be  compared  on  a quantita- 
tive basis.  It  is  therefore  necessary  to  expand  this  expression  further  into 

probabilities  of  the  basic  events  E , E. , . . . , E0.  The  use  of  the  method  of  con- 

g I a 

sensus  to  obtain  these  probabilities  is  discussed  in  Section  5.0.  Substitution 
of  these  probabilities  will  give  us  numerical  values  for  the  probability  of  hazard. 

We  see  that  the  above  expression  is  complicated,  and  further  expansion 
into  the  probabilities  of  the  basic  events  will  give  rise  to  a very  complex  ex- 
pression. It  is  therefore  desirable  to  approximate  the  expressions,  Eq.  6 and  7, 
without  losing  significant  terms. 

For  a grounded  type  lawn  mower,  the  probabilities  of  the  events  E , 

A 

Eg,  and  E^.  are  small,  and  therefore,  a product  of  the  two  or  more  such  terms 
becomes  negligibly  small.  Neglecting  these  product  terms  from  Eq.  6,  we  get 
our  first  approximation  as: 


P(H)  * P(E  )[P(Ea)+P(Eb)+P(Ec)] 


v 
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From  Fig.  B2,  we  see  that  the  event  E occurs  only  when  event  En 

A U 

and  event  E,  occur  simultaneously.  Event  E . therefore  is  the  intersection  of 
1 A 

the  two  events  E^  and  Ej.  Probability  of  the  event  E^  is  thus  given  as: 

P(Ea)  = P(E0-Ej) 

and  assuming  that  events  Eq  and  Ej  are  independent: 

P(Ea)  = P(E0)P(Ej)  (9) 

Similarly,  probabilities  of  events  Eg  and  E^.  can  be  expressed  as 

follows: 

P(Eb)=  p{e2-(E3+E4)} 

= P(E2)P(E3+E4) 

= P(E2){p(E3)+P(E4)-P(E3*  e4)| 

= P(E2){P(E3)+P(E4)-P(E3)P(E4)}  (10) 

P(EC)  =P[E5-E6-(E7+E8)] 

= P(E5)P(E6)P(E7+Eg> 

= P(E5)P(E6)[P(E7)+P(E8)-P(E7  • Eg)] 

* P(  Eg)  P(  E6)  [ P(  E?)  + P(  Eg)  - P(  E7)  P(  Eg)  ] (11) 

Substituting  the  expressions  for  P(EA),  P(Eg),  and  PfE^)  into  Eq.  8, 

we  get  our  first  approximation  as: 

P(H)  = P(Eg){p(E0)P(E1)+P(E2)[P(E3)+P(E4)-P(E3)P(E4)] 
+P(  Eg)  P(  E6)  [ P(  E?)  +P(  Eg)  - P(  E7 ) P(  Eg)  ]} 


f 


I 
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Since  all  the  probabilities  of  basic  events  are  small,  P(E^)  < 1,  we 
can  make  further  approximations.  In  the  above  expression  we  see  that  the 
product  term  P(E3)P(E^)  is  very  small  compared  to  the  sum  term  [P(E^) +P(E^) ] , 
and  the  product  term  P(E^)P(Eg)  is  also  very  small  compared  to  [P(E7)+P(Eg)]. 

We  can  therefore  omit  these  terms  without  the  loss  of  much  accuracy  in  the 
final  value.  The  final  approximation  for  P(H) , therefore,  is: 

P(H)  * F(E  ){P(E0)P(E1)+P(E2)CP(E3)+P(E4)] 

+P(E5)P(E6)[P(E7)+P(E3)]}  (13) 

In  the  case  of  the  doubly  insulated  type  lawn  mower,  following  similar 
arguments,  we  obtain  as  the  final  approximation  for  the  probability  of  hazard 
P(H’) : 

P(H')  » P(E^){P(E^)P(E’1)+P(E^CP(E'3)+P(E^)] 

+P(E’5)P(E^)[P(E^)-P(E^)]}  (14) 

4.  0 Consensus  Approach  to  Estimation  of  Hazard  Probabilities 

4.  1 Basis  of  Consensus:  In  our  Lawn  Mower  Safety  Studies,  hard  quotable  data 
was  not  available;  therefore,  we  adopted  a consensus  approach  to  the  estimation 
of  hazard  probabilities.  The  reasons  for  non-availability  of  hard  data  in  this 
case  are: 

(a)  Improper  or  non-recording  of  hard  data:  In  our  electric  shock  hazard  study 
we  needed  frequency  of  shock  injury,  but  we  could  not  get  such  data  as  it 
is  not  recorded.  The  reasons  for  non-recording  of  such  data  are:  Firstly 

* 

there  is  no  central  agency  which  maintains  records  on  lawn  mower  injuries. 

( R q£ . 9) 

The  NEISS  injury  data  collection  system  * ' run  by  the  U.  S.  Con- 
sumer Product  Safety  Commission  collects  injury  data  by  maintaining  a survey 
at  statistically  selected  hospital  emergency  rooms.  Little  data  exists  on  lawn 
mower  electric  shock  which  is  a few  years  old.  Cases  of  electrocution  or  minor 
shock  discomfort  may  never  pass  through  a hospital  emergency  room. 


> 
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decondly,  for  minor  shocks  people  are  unlikely  to  spend  time  in  reporting 
the  shock.  Thirdly,  in  case  of  severe  injuries,  they  might  not  report 
either  due  to  emotional  upset  or  based  on  counsel  from  the  legal  adviser 
who  would  be  concerned  how  such  a report  might  affect  their  compensation. 

(b)  Proprietary  Data:  In  some  cases  failure  data  is  recorded  by  the  manufac- 
turer, but  because  of  perceived  or  real  competitive  and  legal  pressures, 

he  will  not  permit  the  data  to  be  directly  released  and  attributed  to  the  firm. 
In  many  cases  he  would,  however,  report  such  data  (perhaps  in  summary 
form)  as  an  estimate. 

(c)  Newness  of  Concept  or  Design:  When  some  new  product  is  introduced  (by 
neW  product  we  mean  the  concept  itself  is  new  and  not  just  an  old  product 
introduced  by  a new  company),  it  takes  several  years  before  any  significant 
real  data  is  accumulated. 

4.  Z Knowledge  of  the  Expert:  In  the  consensus  approach  we  ask  a panel  of  ex- 
perts to  give  their  estimates  (guesses)  for  the  basic  probabilities.  The  'cor- 
rectness' of  the  results  depends  upon  the  knowledge  of  the  experts  and  the  size 
of  the  sample  (panel). 

By  knowledge  of  experts,  we  mean  the  experience  and  familiarity  with  the 
particular  problem  and  not  their  general  credentials.  In  our  study  (electric 
shock  hazard)  estimates  from  electrical,  mechanical,  and  industrial  engineers 
would  show  better  results  than  those  from  bank  tellers  and  secretaries.  On  the 
other  hard,  if  we  wish  to  study  availability  (or  reliability)  of  electric  typewriters 
estimates  from  secretaries  and  clerks  would  be  more  valuable  than  those  from 
electrical  engineers. 


/ 
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Members  of  our  panel  were  qualified  engineers  working  in  manufac- 
turing, designing,  management,  or  academic  fields.  In  addition,  many  of 
the  panel  were  also  working  members  of  the  Consumers’ Union  Study  Group 
for  the  development  of  a Safety  Standard  for  Power  Lawn  Mowers.  This  work- 
ing group  was  familiar  with  the  reports  published  by  CPSC,  CU,  NEISS,  etc.  , 
on  the  available  injury  data  concerning  Power  Lawn  Mower  Shock  hazards,  as 
well  as  similar  problems  with  other  electrical  appliances.  In  our  first  estimate 
twelve  experts  participated,  and  seven  of  these  submitted  revised  estimates  after 
seeing  a summary  of  the  original  results. 


i 
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4.  3 Different  Types  of  Consensus:  Clearly,  there  are  several  ways  to  imple- 
ment the  concepts  of  consensus  using  statistical  estimation  principles.  Some 
statisticians  have  proposed  using  Bayesian  Analysis  techniques,  and  the  Appen- 
dix of  this  report  develops  a set  of  techniques  which  are  related  to  classical 
maximum  likelihood  theory.  Although  the  details  differ,  the  end  numerical  re- 
sults should  be  similar. 

When  one  asks  an  expert  for  a probabilistic  estimate,  he  might  request 
a point  estimate,  a range  estimate  (min  and  max),  or  a min,  max,  and  average 
estimate.  In  Section  5 the  panel  of  lawn  mower  experts  were  asked  to  give  a 
point  estimate.  We  then  contemplated  using  either  the  mean,  the  median,  or 
the  maximum  or  minimum  as  our  estimator,  which  is  discussed  in  Section  5. 
Subsequent  to  this  study,  the  authors  concluded  that  a two  or  three  point  estimate 
was  superior,  and  the  estimator  formulas  given  in  the  Appendix  were  generated. 

In  addition  to  the  number  of  points  guessed  by  each  expert,  we  may 
wish  to  rate  the  experts  (either  a self-rating  or  a rating  by  those  who  analyze 
the  data).  If  such  ratings  appear  meaningful,  we  may  wish  to  give  greater  weight 


to  some  experts. 
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In  all  cases  of  interest  the  random  variable  will  be  time  to  occurrence 
of  some  failure  or  unsafe  condition.  Experience  in  the  reliability  field  has 
shown  that  either  exponential  or  "W eibull  distributions  are  the  most  appropriate 
models.  General  formulas  for  any  one  and  two  parameter  distributions,  as  well 
as  specifically  the  exponential  and  W eibull  models,  are  derived  in  Appendix  A. 

4.  4 Analytical  Approach  to  Consensus:  We  have  already  discussed  in  Section  1 
the  several  approaches  one  may  take  to  estimation  of  probabilities  using  consensus. 
Appendix  A contains  a mathematical  development  of  each  of  these  methods.  We 
will  summarize  the  results  only  in  this  section,  and  we  refer  the  reader  to  Ap- 
pendix A for  details. 

If  we  have  a single  expert,  we  could  generate  from  one  to  three  esti- 
mates depending  on  how  he  is  instructed.  If  asked  to  give  one  best  guess,  only 
one  data  point  would  be  obtained.  However,  if  he  were  asked  to  guess  the  mini- 
mum and  maximum,  two  data  points  would  result.  Similarly,  if  he  is  asked  for 
a minimum,  maximum,  and  mean,  three  data  points  result.  We  consider  the 
two  and  three  data  point  cases  in  Appendix  A and  discuss  the  single  data  point 
case  in  Section  5 (for  n experts).  If  the  underlying  failure  density  is  assumed  ex- 
ponential (constant  hazard),  then  the  single  failure  rate  parameter,  \,  and  its 
variance  is  estimated  using  formulas  A16-A19.  Similar  expressions  for  a W eibull 
density  function  (hazard  = ktm)  are  derived  in  Eqs.  A21-A28.  In  this  case  we 
must  estimate  both  the  scale  parameter  k and  the  shape  parameter  m and  their 
variances. 

For  the  case  of  n independent  experts,  these  expressions  generalize 
(as  intuition  would  predict)  to  Eqs.  A29-A36.  Note  that  the  exponential  case  can 
be  considered  a special  case  of  the  Weibull,  where  m = 0,  and  k a \. 

r 
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As  was  previously  discussed,  we  may  not  wish  to  give  all  experts' 
guesses  the  same  weighting.  In  this  case  a simple  approach  is  to  use  the 
equations  cited  above  for  the  single  expert  case  and  obtain  a set  of  estimates 
for  each  expert.  These  can  then  be  combined  via  the  linear  combination  given 
in  Eq.  A37.  If  we  wish  the  estimate  to  be  unbiased,  the  weighting  factors  must 
sum  to  unity  as  in  Eq.  A38.  If  we  wish  to  use  experience  and  judgment  to  assign 
the  relative  weights  to  the  various  a's,  then  the  form  given  in  Eq.  A39  is  more 
convenient.  On  the  other  hand,  if  we  wish  the  variance  of  the  estimator  to  be 
minimized,  then  the  weighting  coefficients,  a^'s,  are  inversely  proportional  to 
the  variance  of  the  individual  guesses  as  shown  in  Eq.  A40. 

5.  0 Results  of  the  Lawn  Mower  Shock  Hazard  Study 

5.  1 Introduction:  As  discussed  previously,  while  the  mathematical  basis  of 
consensus  estimation  was  being  explored,  in  parallel  the  concepts  were  being 
applied  in  the  Lawn  Mower  Safety  Study.  Because  of  circumstances,  much  of 
the  application  preceded  the  theoretical  studies.  Thus,  in  this  section  we  deal 
with  a single  estimate  from  each  of  the  experts  and  use  the  median  of  the  distribu- 
tion as  our  parameter  estimator. 

5.  Z Consensus  Estimates:  Early  in  the  discussions  of  the  CU/CPSC  Electric 
Shock  Hazard  Subcommittee,  we  decided  that  it  would  be  a useful  learning  exer- 
cise to  apply  Fault  Tree  Analysis  and  Consensus  Estimation  to  the  quantitative 
analysis  of  lawn  mower  shock  hazards.  In  addition,  the  group  felt  it  would  be  of 
use  in  comparing  the  shock  hazards  due  to  grounded  type  and  doubly  insulated  type 
electric  lawn  mowers. 

Initially  twelve  experts  participated  in  the  estimation  process.  After 
receiving  a summary  of  the  first  round  of  estimation,  and  a discussion  of  a 
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subcommittee  meeting,  the  group  was  asked  to  send  in  revised  estimates,  and 
seven  responded. 

On  the  second  round  the  experts  were  asked  to  estimate  the  probabili- 
ties associated  with  present  lawn  mowers,  as  well  as  how  they  would  change  if 
the  proposed  safety  standard  were  implemented  in  the  form  envisioned  at  that 
time.  Not  all  the  respondents  gave  both  estimates  on  the  second  round. 

Although  all  the  estimates  were  shown  individually  to  the  Committee, 
they  were  coded  by  number  (only  known  to  Mr.  Sinkar,  who  did  not  participate 
as  an  expert),  and  thus  personalities  had  little  influence  on  the  consensus  process. 

5.  3 Numerical  Probabilities:  We  received  twelve  responses  from  the  experts 
for  our  first  iteration.  All  the  participants  gave  two  sets  of  basic  probabilities 
(cf.  Figs.  B2  and  B3),  one  for  the  grounded  type  lawn  mower  and  the  other  for 
the  doubly  insulated  type.  (See  Appendix  C,  Tables  Cl,  C2).  All  the  estimates 
were  without  consideration  of  the  CPSC  mandatory  standard. 

We  preferred  the  median  method  to  the  mean  method  because  in  the 
mean  method  one  or  two  estimates  of  extremely  high  (or  low)  numerical  value 
tend  to  shift  the  results,  whereas  in  the  median  method  it  is  the  distribution  that 
plays  the  important  role.  (See  Tables  C3,  C4.  ) To  illustrate  this  further, 
we  cite  an  example  from  "Project  DELPHI"/  ^ 

The  experiment  was  designed  to  apply  expert  opinion  to  estimate,  from 
the  viewpoint  of  a Soviet  strategic  planner  who  is  devising  an  optimal  attack  on 
U.  S.  industry,  the  number  of  A-bombs  required  to  reduce  the  munitions  output  by 
a prescribed  amount. 

A further  discussion  of  the  relative  merits  of  the  mean  and  the  median 
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is  given  in  Section  6. 
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The  primary  estimate  (first  iteration)  received  from  seven  respondents 
are  as  follows: 


j Respondent  1 

2 

3 4 5 6 7 

j Estimated 

Number  125 

1 

50 

150  300  200  1000  5000 

The  mean  of  the  above  estimates  is  975.  We  clearly  see  that  the  mean  shifted 
by  two  high  estimates,  i.  e.  , 1000  and  5000.  The  median  of  the  above  estimates 
is  200.  We  also  see  that  four  out  of  seven  estimates  lie  in  the  small  range  of 
100-300.  Hence  the  median  is  a more  representative  estimate  than  the  mean. 

In  our  lawn  mower  study,  after  substituting  the  median  probability  of 
each  of  the  basic  events,  from  Tables  C3  and  C4  we  get  the  following  numerical 
values  for  the  probabilities  of  the  hazards: 

P(H)  = 0. 554265625  x 10‘2 
P(  H')  =0.  2361775  x 10‘2 

We  then  fed  back  and  discussed  the  summary  of  the  results  with  the 
experts  and  requested  the  revised  estimates.  This  time  we  requested  two  sets 
of  estimates  for  each  type  of  lawn  mower.  The  first  set  consisted  of  the  numeri- 
cal probabilities  without  consideration  of  the  mandatory  standard  and  the  second 
set  with  consideration  of  the  mandatory  standard. 

This  time  we  received  seven  estimates  of  probabilities  with  considera- 
tion of  the  mandatory  standard  and  three  estimates  without  consideration  of  the 
mandatory  standard.  (See  Tables  C5-C10.) 

After  substituting  the  medians  for  each  of  the  basic  probabilities,  we 
get  the  following  second  round  numerical  values  for  the  probabilities  of  the  hazard. 
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Without  safety  mandate: 

P(H)  = 3.  1875  x 10~2 
P(H')  = 0.  805005  x 10' 2 

With  safety  mandate: 

P(H)  = 0. 08064  x 10’2 
P(H')  = 0.  08033296  x 10'2 

5.  4 Conclusions:  Although  we  cannot  draw  general  conclusions  as  to  convergence 
or  validity  of  consensus  probabilities  from  one  such  an  experiment,  we  can  cite 
the  following  conclusions  pertaining  to  this  particular  problem: 

(1)  The  Fault  Tree  representation  gave  better  insight  to  the 
different  faults  and  their  effects  on  the  hazard.  It  also  helped 
in  assuring  that  no  important  possible  fault  was  left  out. 

(2)  The  second  round  of  the  estimates  gave  a smaller  range  within 
which  all  the  estimates  lay,  although  we  could  not  draw  any  con- 
clusions about  their  distribution. 

(3)  Without  safety  mandates  the  probability  of  shock  hazard  of  the 
doubly  insulated  lawn  mower  is  lower  (25%)  than  the  probability 
of  the  shock  hazard  of  the  grounded  type  lawn  mower.  After 
introduction  of  the  safety  mandates,  the  probabilities  of  the 
shock  hazards  of  both  the  types  is  approximately  the  same. 

The  general  opinion  of  the  members  of  the  sub-committee  was  that  al- 
though the  validity  of  the  exact  numerical  values  of  the  probabilities  could  be 
questionable,  the  analysis  provided  relative  rankings  which  helped  in  establishing 
the  safety  mandates. 
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6.  0 Preliminary  Analyses  of  Raw  Consensus  Data 

In  reviewing  an  early  version  of  this  material,  Mr.  George  Leven- 
bach^Ref‘  called  to  the  authors'  attention  the  fact  that  there  might  be  signifi- 
cant bias  in  consensus  data,  and  it  would  be  worthwhile  to  consider  preanalysis  of 
this  data.  F or  example,  it  might  quite  likely  be  that  among  our  group  of  experts 
there  would  be  a cluster  of  optimists  and  a cluster  of  pessimists,  who  fairly  con- 
sistently estimated  probabilities  high  or  low  depending  upon  this  particular  bias. 
Levenbach  suggested  that  we  apply  some  of  the  new  statistical  techniques  of 
cluster  analysis  and  spanning-tree  analysis.  *3) 

Since  many  of  the  readers  may  be  unfamiliar  with  this  new  body  of  sta- 
tistical techniques,  (as  were  the  authors  before  Levenbach  called  it  to  their  at- 
tention) a simplified  heuristic  technique  is  us^d  in  this  paper  to  perform  in  essence 
the  same  type  of  analysis  of  bias.  Additional  approaches  are  discussed  in  Sections 
6.  2 and  6.  3. 

6.  1 Geometric  Approach:  The  heuristic  approach  assumed  that  there  might  be 
two  homogeneous  clusters  (or  only  one).  The  highest  and  lowest  estimates  were 
located  (in  some  cases  an  obvious  outlier  was  dropped)  and  the  geometric  mean 
used  as  the  dividing  line.  The  upper  group  (pessimist)  was  separately  plotted  as 
was  the  lower  group  (optimist).  If  the  resulting  distributions  appeared  clustered, 
then  the  two  cluster  hypothesis  was  accepted,  and  the  data  analyzed  separately. 

If  not,  then  the  hypothesis  that  no  clustering  is  present  is  accepted, and  the  data 
is  lumped  for  analysis. 

In  applying  this  geometric  technique,  we  plotted  a frequency  distribution 
of  the  responses  on  semilog  paper  to  accommodate  the  wide  dynamic  range  of  the 
probabilities.  The  probability  values  were  plotted  on  the  abcissa  (the  log  scale) 
and  the  coded  number  (so  we  could  trace  the  expert's  identity)  was  plotted  on  the 
ordinate.  See  Figure  B5  which  is  one  of  the  twenty  such  figures  which  could  be 
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drawn  from  the  data  in  Tables  Cl  and  C2.  The  reader  will  observe  that  the 
logarithmic  scale  has  nicely  clustered  the  data. 

The  use  of  the  geometric  mean  as  a dividing  point  is  essentially  the 
same  as  deciding  to  use  the  arithmetic  mean  of  the  logarithm  of  the  values. 

Geometric  Mean  = GM  = (x^  x^)1/2  (15) 

log(GM)  = ^ (log  xL  + log  xj  (16) 

If  the  expert  guess  was  above  the  GM,  then  we  called  him  a pessimist, 

P,  and  if  below,  we  called  him  an  optimist,  O.  Tables  Cll  and  C12  present  the 
results  for  all  the  experts  on  the  ten  probabilities  which  were  estimated  for  the 
grounded  type  model.  The  last  column  represents  the  number  of  times  he  was 
judged  a pessimist  divided  by  10,  which  is  his  pessimist  rating.  Inspection  of 
this  column  shows  that  all  experts  except  number  5 were  quite  consistently  either 
an  optimist  or  a pessimist.  Thus  the  data  was  considered  to  be  clustered  and  a 
high  median  and  a low  median  was  computed  in  each  case  as  given  in  Table  Cl  3. 

2 Average  Distance  Method:  In  this  method  we  classify  estimates  in  the  same 
cluster  if  they  are  sufficiently  close  to  one  another.  The  measure  of  relative 
closeness  to  be  used  will  be  clarified  by  an  example.  Consider  the  eight  data 
points: 

.0001,  .00015,  .0002,  .05,  0.75,  0.8,  0.82,  0.9 

If  we  plot  these  graphs  and  study  the  results,  our  visual  perception  would  classify 
them  in  two  clusters,  i.  e.  , cluster  1 (.  0001 , .00015,  . 0002)  and  cluster  2 (0.  75, 
0.8,  0.82,  0.9).  We  would  include  in  cluster  2 points  at  0.  82  and  0.  9 although 
they  are  0.08  units  apart.  At  the  same  time  we  would  be  hesitant  to  include  the 
unclustered  point  . 05  in  cluster  1 although  it  is  only  . 0498  units  from  the  next 
smallest  value.  This  is  because  the  distance  though  small  in  absolute  value,  is 
much  larger  (over  900  times)  than  the  distance  between  any  other  two  points  in 
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cluster  1.  Thus  unless  the  point  at  0.  05  is  to  be  treated  as  a third  cluster,  it 
is  better  to  lump  it  with  cluster  2,  where  its  distance  from  the  rest  of  the  group 
is  only  about  three  times  the  next  largest  spacing. 

Since  we  wish  to  deal  with  ratios  of  values,  we  deal  with  logarithms 

of  the  values,  thus  we  prepare  a table  showing  logarithm  and  coded  number  for 

each  expert  in  ascending  order  of  log  probability.  The  last  column  in  the  table 

represents  the  distance  of  the  point  from  the  next  higher  value.  Tables  Cl 5 to 

C34  summarize  the  results  for  all  the  events  (i.  e.  , E , . . . , En;  E'  , . . . , E'Q)  of 

g 8 g 8' 

the  study.  Obvious  outliers  were  neglected  from  our  computation.  We  then  com- 
puted average  distance  between  two  points.  'When  two  (or  more)  experts  guessed 
the  same  probability  value  (and  hence  it3  logarithm),  we  considered  them  as  two 
(or  more)  values  separated  by  zero  distance  for  our  computation  purpose.  W e 
defined  our  separating  distance  equivalent  to  three  times  the  average  distance. 

That  means  if  the  distance  between  any  two  consecutive  points  is  greater  than 
three  times  the  average  distance,  then  the  two  points  belong  to  two  separate 
clusters,  otherwise  they- belong  to  one  cluster. 

Tables  C15  to  C34  3how  that  out  of  twenty  events  (E  , ....  E0;  E'a), 

g So 

four  events  (Ey  E-,,  E' and  E'^)  give  one  cluster,  two  events  (E^  and  E'5)  give 
three  clusters,  and  the  next  fourteen  events  give  two  clusters. 

It  is  interesting  to  see  that  the  results  of  these  fourteen  (two  cluster) 
events  are  the  same  as  the  Geometric  Mean  method  except  for  four  events.  In 
the  case  of  event  Eq  expert  10  is  classified  as  a pessimist  rather  than  an  optimist 
in  the  G.  M.  method;  in  case  of  event  E^,  expert  5 is  classified  as  a pessimist 
as  against  optimist  in  the  G.  M.  method;  in  case  of  event  Eg,  expert  9 is  classified 
as  pessimist  as  against  optimist  in  the  G.  M.  method;  and  in  case  of  event  £^, 
expert  9 is  classified  as  optimist  as  against  pessimist  in  the  G.  M.  method. 

There  are  some  obvious  advantages  and  disadvantages  of  this  method 
over  the  heuristic  (G.  M.  method).  The  heuristic  method  always  gives  two  and 
only  two  clusters,  but  the  average  method  may  give  one,  two,  three,  or  sometimes 
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more  than  three  clusters.  In  our  analysis  we  can  only  use  up  to  three  clusters. 

(One  cluster  would  give  a point  estimate,  two  clusters  would  give  maximum  and 

minimum,  and  three  clusters  maximum,  minimum,  and  most  likely.)  The  second 

disadvantage  is  that  there  is  no  uniformity  in  the  number  of  clusters  (i.  e.  , for 

some  events  (e.  g.  , E3)  we  get  one  cluster,  for  some  (e.  g.  , Ej)  we  get  three 

clusters).  In  our  analysis  we  require  to  combine  all  the  probabilities  (E  , . . . , E ) 

g 8 

to  compute  the  probability  of  the  top  event  (i.  e.  , electric  shock).  Under  such  a 
condition  there  is  no  simple  way  of  combining  these  events.  One  way  to  overcome 
this  difficulty  is  to  divide  the  guesses  at  the  points  between  which  the  distance  is 
maximum.  This  procedure  will  give  two  clusters  except  when  the  maximum  dis- 
tance is  identical  for  more  than  one  pair  of  points. 

The  major  advantage  of  the  average  distance  method  is  that  the  de- 
cisions on  grouping  are  based  on  the  relative  distance  between  points,  rather 
than  the  geometric  mid-point  between  the  extremes. 

8-  3 Correlation  Clustering:  Another  clustering  technique  can  be  used  which  allows 
us  to  control  the  number  of  clusters  and  also  utilize  information  on  all  the  inter- 
point distances,  not  just  the  adjacent  distances.  This  is  based  on  an  adaption  of 
the  method  of  Fortier  and  Solomon^  ^ . We  will  not  completely  describe 

this  method  but  will  sketch  its  features  and  application  to  our  consensus  estima- 
tion method.  An  outline  of  the  method  follows: 

(1)  One  uses  the  correlation  coefficient  p as  a measure  of 
the  distance  between  two  variables, 

(2)  The  B function  ("belongingness"  function)  is  computed  as 
the  ratio  of  the  average  p for  pairs  belonging  to  the  same 
cluster  to  the  average  p for  pairs  not  belonging  to  the  same 
cluster, 
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(3)  The  B function  is  computed  for  all  possible  partitions, 
and  the  optimal  cluster  configuration  is  chosen  as  that 
one  with  the  largest  B. 

The  procedure  becomes  intractable  rapidly  as  the  number  of  points  and  the 
corresponding  possible  partition  combinations  increase.  In  fact,  Fortier  and 
Solomon  focus  on  sampling  methods  to  provide  a tractable  approximate  solution. 

In  our  consensus  method  we  can  considerably  restrict  the  problem 
in  that  we  are  only  interested  in  really  the  best  two  clusters  or  best  three  clusters. 
If  we  view  our  set  of  expert  guesses  as  n points  plotted  along  the  x axis,  it  is 
easy  to  see  that  there  are  only  n-1  possible  dividing  lines  (clusters).  We  can 
develop  the  equivalent  formula  for  three  clusters  by  the  following  procedure. 

Divide  points  1 and  2 to  form  2 clusters.  Then  there  are  n-2  ways  of  separating 
the  remaining  clusters.  Now  consider  the  first  dividing  line  between  points  2 
and  3.  Now  there  are  n-3  ways  of  separating  the  remaining  clusters. 

In  computing  the  number  of  clusters  up  to  three,  N^,  the  degenerate 
cases  of  only  1 and  2 clusters  appear  as  the  first  two  terms  and  the  summation 
term  includes  all  the  partitioned  3 cluster  cases. 

n-2  n-1 

N3  = 1 + (n-1)  + 

i=2  i=  1 

n-1  n-1 

_V  V • _ / n n(n-l)  _ n(n-l)  ,,,, 

L n " L 1 = n(n_1)  ■ (Y) 

i=  1 i-1 

Thus,  even  for  3 partitions  the  number  of  trials  is  about  n /2.  Since  it  is 
unlikely  that  n will  be  much  larger  than  20  or  30  at  most,  perhaps  a thousand 
computational  cases  are  required,  which  is  certainly  feasible.  The  reader  is 
referred  to  Ref.  (14)  for  further  details  of  correlation  clustering  method. 
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A.  1 Introduction 

The  mathematical  development  which  follows  can  be  derived  inde- 

(Ref  1 5) 

pendently  or  can  be  related  to  order  statistics  ' ' where  the  sample  size 

is  2 or  3.  The  assumption  is  that  k experts  independently  supply  minimum  and 
maximum  or  minimum,  maximum,  and  average  estimates  of  the  random  vari- 
able time  to  failure.  The  joint  distribution  of  the  lower  estimate  and  upper 
estimate  (or  the  lower  estimate  and  the  range)  ie  derived,  based  upon  an  assumed 
sampling  density  function  chosen  from  engineering  experience  or  prior  data. 

The  estimates  of  the  k experts  are  then  combined  using  a choice  of  techniques; 

(1)  Maximum  likelihood  estimation  (MLE)  theory, 

(2)  An  unbiased  linear  combination  (ULC)  of  the  k estimates 
where  the  weighting  coefficients  may  be  chosen  according 
to  the  confidence  we  place  in  each  estimation, 

(3)  Using  a linear  unbiased  minimum  variance  (LUMV) 
estimator.  In  this  case  the  expert  who  guesses  the 
smallest  range  has  the  most  influence. 

The  analyst  is  free  to  pursue  whichever  of  these  approaches  he  prefers.  Lacking 
any  detailed  data  about  the  relative  experience,  conservatism,  biases,  or 
psychological  factors  which  might  underlie  the  experts'  guesses,  the  MLE  ap- 
proach would  be  preferred. 

A.  2.  Derivation  of  Joint  Density  for  Two  Guesses 

Suppose  that  each  of  the  k experts  guesses  a minimum  and  maximum 
value  for  the  random  variable,  time  to  failure  (occurrence).  For  the  first  ex- 
pert our  notation  will  be  t.  . and  t,  and  for  the  k'th  expert  t,  . and  t, 
r 1 min  lmax  r kmin  Km  ax 

for  the  random  variables,  and  x.  and  x.  for  the  numerical  values  of  the 

Tcm  in  Km  ax 

guesses  for  the  k'th  expert.  The  joint  density  function  for  the  k'th  expert  may 


A-2 


I 


be  derived  as  follows.  Letting: 

^Xkmin  Vmin  < ^kmin  + Axmin’  \rnax  £ ''kmax  ^ ^max  + Axmax^ 


The  joint  probability  of  the  min  and 
max  estimates 


= $ (t.  . , t.  )Ax  . Ax 

kmin  km  ax  min  max 


(Al) 


If  we  let  f(t^.)  be  the  sampling  density  function  and  if  both  the  minimum 

and  maximum  estimates  are  independent,  then  since  there  are  two  guesses 

(chances)  that  ^ will  fall  in  the  min  interval,  and  given  this  occurrence  only 

one  chance  that  t,  will  fall  in  the  max  interval  we  obtain: 
km  ax 

-'^tkmin'  tkmaLX^Xmin‘^Xmax  ” 2^tkmin^Xtmin^tkmax^Xr 


max 


and  finally: 


" ^hcmin’ hcmax^  ^ hcmin^^max^ 


For  notational  convenience  we  call  t, = t^  and  tkmax  = t^2 


(A2) 

(A3) 


kmin 


Thus: 


^kl'  tk2)  = 2f(tkl)f(tk2)  (A4) 

A.  3.  Derivation  of  the  Joint  Density  Function  for  Three  Guesses 

We  can  derive  the  joint  density  function  for  three  guesses  in  a similar 
manner.  We  let  t^.^  t^,  and  t^^  be  denoted  by  t^,  t^.  and  t^  re- 
spectively and  we  again  assume  independence.  There  are  three  ways  the  first 
interval  can  be  filled,  then  given  it  is  filled  two  ways  of  filling  the  second,  and 
given  that  the  first  and  second  are  filled  one  way  of  filling  the  third.  Thus: 


<*)<tkr  *k2'  tk3)  = t 3^tkl^x^2f^tk2^x[^tk3)^ 
* 6f(tkl)f(tk2)f(tk3) 


(A5) 


> 
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A.  4.  Estimation  of  Density  Function  Parameters  for  One  Expert 

We  assume  that  the  density  function  depends  on  either  one  parameter 
0,  or  two  parameters  9^  and  0^,  i.  e.  , f(9^)  or  f ( 0 ^ , Q^).  Using  maximum  likeli- 
hood theory  for  a single  parameter  density  function,  we  obtain  the  likelihood  function 
from  the  joint  density  Eq.  A4. 

L(tkl’  ^2'  91)  = 2f(tkl)f(tk2)  (A6) 

The  MLE  estimate  for  9^  is  found  by  setting: 

- n 


59 


As  a convenience  in  the  differentiation  one  generally  takes  the  In  of  L 
first.  This  is  valid  since  the  maximum  of  InL  and  L occur  at  the  same  point  due 
to  the  monolithicity  of  the  In  function.  The  labor  saved  in  dealing  with  logarithms 
is  due  to  the  conversion  of  a product  form  into  a sum  form,  which  is  easier  to 
differentiate. 


In  L = £ 


. M = 0 

50! 


(A7) 


£ = ln2  + 1 


nf^j)  + lnf(tk2) 


5£  _ 
59, 


5 lnf(tkl)  5 lnfft^.,) 


5 9 


1 


5 0 


(A8) 


1 


Similarly,  for  three  estimates  we  use  Eq.  A5  and  obtain: 

b£  _ & 1 H^kl^  ± & lnf(tk2)  , 6 lnf(tk3^_  A 


5 9 


50, 


5 9, 


<A9) 
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In  the  case  of  two  parameters  we  obtain  (analogous  to  Eqs.  A8  and  A9): 


& lnf(tkl)  + 5lnf(tk2>  = Q 


be 


b9, 


b 0, 


b£  = b lnf(tkl> 

be2  b e2  b e2 

and  for  three  estimates  and  two  parameters  we  obtain: 


(A10) 


(All) 


b£ 

b9, 


b ^(^^  & 1 ^(t^.^)  blnf(tk3) 


b 0 


b 9 


1 


b 0 


1 


St  Slnflt^)  | a Infill  t Infill 
b9->  b 9->  b 0?  b 9o 


(A12) 


(A13) 


32  a °2  0 °2  u °2 

Of  course  in  the  case  of  Eqs.  A10,  All  and  A12,  Al  3 we  must  solve  the  two  re- 
sultant equations  simultaneously  for  0^  and  9^.  Since  these  equations  are  often 
nonlinear  and  transcendental,  a numerical  solution  is  often  required.  Further 
discussion  of  numerical  solutions  appears  in  the  references.  w e can 

also  estimate  the  variance  of  our  estimate  of  9 (note  the  "hat"  notation  above  0 

(Ref  17) 

means  an  estimate  of  9)  by  using  the  large  sample  limiting  form.  ' * The 

computation  includes  taking  the  second  derivative  of  the  In  of  the  likelihood 
function  and  substituting  in  the  expression: 


Var  9 at 


1 

TV 

bV 


(A  14) 


A.  5 Estimators  of  Exponential  Density  Function  Parameters  for  One  Expert 

We  now  assume  that  experience,  engineering  assumption,  or  similarity 
between  the  event  in  question  and  other  known  data  lead  the  analyst  to  conclude 
that  the  hazard  function  for  the  data  is  constant,  i.  e.  , the  density  function  is 
exponential: 


A-5 


f(9)  = f(X)  = \e'U 


(A15) 


Substitution  in  Eqs.  A8  and  A14  as  well  as  differentiation  and  simplifi- 

A 

cation  and  subsequent  substitution  of  \ for  X in  the  variance  expression  yields: 


X = 


tl  + h 


t X 

var  X 

Similarly  for  three  estimates  we  obtain: 


(A16) 

(A17)* 


X = 


tl  + t2  + t3 


(A18) 


var 


X 


(A19)* 


The  results  for  the  exponential  density,  Eqs.  A16-A19,  are  just  what  intuition 
would  lead  us  to  believe. 


A.  6 Estimators  of  Weibull  Density  Function  Parameters  for  One  Expert 

If  we  suspect  that  wearout  is  present  or  there  are  other  good  reasons 
to  believe  that  the  hazard  function  is  not  constant,  then  we  can  use  the  Weibull 
model.  The  Weibull  hazard  ktm  corresponds  to  the  density  function: 


f(k,  m)  = ktm e 


-(• 


k N ,m+l 


(A20) 


In  an  analogous  fashion  to  the  work  of  the  previous  section^ e^‘  the  likeli- 
hood equations  for  an  expert  and  two  estimates  become: 


These  are  rough  approximations  as  are  Eqs.  A2  3,  A24,  A27,  A28 
since  the  sample  size  is  2 or  3.  However,  in  Section  A.  7 the  sample  size  is 
2n  or  3n  and  the  approximation  will  be  better. 

r 


C - 2(m+l) 

m+1  , m+1 

hcl  + tk2 

i 

Simultaneous  solution  requires  either  a graphical  solution  or  an 
( R ef  19) 

computer  solution  ' . In  order  to  compute  the  variance: 

+ 2 

Var  k » 


2IC‘ 


2 

m+1 


I £+‘ 


Var  m as 


(m+l) 


2 

\ m+1 
3Z-Td 
1 


1/k 


(m  + 1) 


' m+1, 

ihd 

i 


1 V .m+1,,  ,2 

m+1  Z.  tti  (lntki) 

1 


Similarly,  for  one  expert  and  three  estimates  we  obtain: 


k =TJl£2±iL 


I <SW 


I 


A 

(A21) 


(A22) 


iterative 


(A2  3) 


(A24) 


(A25) 
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3 


1 


, m + 1 , 

3 ) t,  . In  t, 

_ K1  xi 

L 


m+1 

L.  "ki 
1 


Var  k 


=S 


k 

T 


(A26) 


(A27) 


Var  m » 


1/k 


-(A  28) 


V.m+1 


(m+1) 


3 zL 


(m+1) 


V .m+1 

2 Lhd 

1 


In  t, 


rCl 


mri 


Z «+l 

1 


(law‘ 


A.  7 Combining  the  Estimates  of  n,  Experts  Using  MXE  Theory  * 

The  likelihood  function,  Eq.  A6,  was  constructed  by  assuming  inde- 
pendence of  the  two  (or  three)  estimates  of  a single  expert.  If  we  have  n 
experts  and  assume  independence,  then  the  likelihood  function  L is  simply  the 
product  of  each  expert's  likelihood  function,  and  the  logarithms  of  the  likeli- 
hood functions  become  additive.  Thus  the  derivative  of  the  sum  is  the  sum  of 
the  derivatives.  Thus,  for  n experts  the  equations  corresponding  to  A.  21 
through  A.  24  become: 


£ 3 2n(m  + l) 


i i <r 

i=i  j=i  - 


(A29) 


Note  that  a conflict  in  notation  exists  between  the  number  of  experts,  k, 
one  of  the  parameters  appearing  in  the  Weibull  model,  Eq.  A20,  which  is  also  k. 
Therefore,  the  number  of  experts  will  be  represented  by  n in  the  remainder  of 


and 


the  text. 


la  t.  . 
ij 


a 2 

2a  7 tm+1 

L L.  ij 

i - _ i=l  j=l 

la  t. . = ' 

lj  a a 

i=  1 j=  1 T”  \ m+1 

L L ij 

i=l  j=l 


I I 


2 n. 
m+1 


(A30) 


Var  k w 


k2 

Tn 


(A31) 


Var  an  a> 


1/a 


a 2 a 2 

2 \ V m +1  2 V V m + 1 


a 2 


) t.. 


(m+l)J  .^i  1J  <»+!><££ 


2 L A tij1  + 1 U Cij  + mTl  I I ‘t”+1(ln  V 


• i=  1 j=  1 


Similarly,  Equations  a25  through  a28  generalize  to  yield: 


(A32) 


£ _ 3 n(  m + 1 ) 
a 3 


y 

V 

m + 1 

L 

L> 

C • • 

ij 

i=ij=i 

n 

3 

- r 

r* 

\ 

m +1 

a 3 

3nz, 

L 

t. . 

IJ 

y y in  t. . 

i=  1 

J =1 

L L ij 

n 

a 

i=l  j-1 

I 

i 

tm+l 

ij 

i=i 

j-i 

j- 

3a 
m + 1 


(A33) 


(A34) 
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Var  k (A35) 

A 

Var  m s» 


(A36) 

A.  8 Combining  the  Estimates  of  n Experts  Using  ULC  Theory 

A 

Assume  chat  we  have  an  estimate  6 for  each  parameter  and  its  variance, 
Var  0,  obtained  by  the  methods  of  Sections  A. 4 to  A.  6.  We  wish  to  weight  the 

A ^ .A  , 

estimates,  0^,  07,  0 y etc.,  since  we  respect  some  experts  opinions  more 
than  others.  Thus  we  let: 

■ 0 = parameter  estimate  derived  for  expert  No.  i (i  = 1 to  n)  and  write 
the  linear  form: 

®ULC  = alel  +a2e2  +---  + an9n  (A37) 

Assuming  that  the  0's  are  independent,  have  the  same  means,  and  requiring 

them  to  be  unbiased,  we  obtain^6^" 

a 

l a.  = 1 (A38) 

i 

We  now  assume  that  the  a.'s  will  be  chosen  so  that  they  reflect  the  relative 
confidence  the  analyst  has  in  each  expert. 

For  convenience  it  is  easier  to  assign  a set  of  weights,  b.,  such  that 
each  b lies  between  1 and  10,  where  the  larger  the  value  of  b^,  the  greater  the 
trust.  Thus,  Eq.  A37  can  be  written  as  : 


1/ n 


(m 


n 3 

| ~ m + 1 
_i  \ ^ U,  L ij 
' i=lj=l 


(m+1)  ‘ 


n 

r 

1 


3 

r 

L 


i=l  j*l 


t.. 

ij 


In  t. . 
ij 


1 

m+1 


/ 


i=  1 i=l 


tu+1(la  V 
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eULC 


( A39) 


A.  9 Combining  the  Estimates  of  n Experts  Using  LUMV  Theory 

We  proceed  as  in  Section  A.  8 and  obtain  Eq.  A30.  We  then  further  re- 
quire that  the  variance  be  minimized.  The  variance  equation  is  given  by 


Var(0ULC)  = alCTl  + a2~2  +-  • * + 4"n  <A4°) 

To  do  this  we  must  use  Lagrange  multipliers  or  substitute  Eq.  A38  into 
Eq.  A40  before  we  differentiate  and  set  equal  to  zero.  Using  Lagrange  multi- 

* * A 

pliers,  computing  , . . . , M and  setting  each  partial  derivative  to 

6aj  ?>a2  ban 

zero  we  obtain  a set  of  simultaneous  equations.  Solution  of  these  equations 
yields: 


a 


n 


Var  (0„) 
n 


I 


1 

Var  (0^) 


(A41) 


Flying  Objects 

Noise  (Hearing) 

Fire  (Gas  or  Electric) 
Limbs  Touch  Blade 

Electric  Shock  (Gas  or  Electric) 

Burns 
Explosion 
Recoil  Starter 


Figure  Bl.  High  Level  Fault  Tree  for  Lawn  Mower 


OPERATOR 


Figure  B2.  Shock  Fault- Tree  of  a Grounded  Type  Lawn  Mower. 


B4 


a.  Considered  to  constitute  double  insulation 
Armature  conductor  Slot  liner  (two  layers) 


b.  Not  considered  to  constitute  double  insulation 


Figure  B4.  Illustrations  Explaining  Double  Insulation*. 





*Ibid, 


Number  of  Responses  Number  of  Responses  Number  of  Responses  Number  of  Responses 


Median  = 0.001 


3«a  «9 

; 

o 5. oOOOl 


2. 

!•  *7 


Probability  of  E| 


Median  =0.2 


8 6 5 9 

• • • • 

I 

0.  0001  0.001 


0.01  0.1 

Probability  of  Ei 


Median 


0 0.00001  0.0001 

0.001  0.01 

0.  1 

1.0 

Probability  of  E' 

4 - 

Median  =0.01 

0 0.00001  0.0001 


0.001  0.01 

Probability  of  E^ 

Figure  B5.  (continued) 
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Probability  of  Eg 
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0.001 


Probability  of  E/ 


6 *9 


0 0.000001  0.00001  0.0001 


3*  *2 


0.  001 


Probability  of  Kl, 


0 0.0000001  0.000001  0.00001  0.0001 


Probability  of  Eg 
Figure  B5.  (continued) 
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TABLE  C-3 

PROBABILITY  SUMMARY  FOR  TABLE  €-1 
(First  Round) 


Event 

Lowest 

Estimate 

Highe  st 
Estimate 

Median 

E 

g 

0.  005 

1.  0 

0.  5 

E0 

0.  0001 

0.  878 

0.  095 

E1 

0 

0.  5 

0.  001 

E2 

0.  001 

1.  0 

0.  5 

E3 

0.  0001  0.7 

' ! 

0.  0113 

E4 

0.  00007 

0.  1 

0.  01 

E5 

0.  0001 

1.  0 

0.75 

E6 

0.  0015 

0.  75 

0.  275 

E7 

0.  000002 

0.  1 

0.  0015 

E8 

0.  000002 



0.  05 

0.  00015 
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TABLE  C-4 


PROBABILITY  SUMMARY  FOR  TABLE  C-2 
(First  Round) 


Event 

Lowest 

Estimate 

E' 

g 

0.  00005 

Ef 

0 

0. 00001 

Ei 

0 

Ez 

0.  0005 

E3 

0 

E4 

0 

E5 

0.  0005 

E 6 

0 

E7 

0 

Highe  st 

Estimate 

Median 

0.  9 

0.  5 

0.  878 

0.  0?  25 

0.  3 

0.  001 

1.  o 

0.  2 

0.  7 

0.  01 

1.  o 

0.  7 

0.  1 

0.  001 

0.  1 

0.  001 

E 8 


0 


0.  05 


0.  0015 
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TABLE  C-5 

ESTIMATED  PROBABILITY  OF  THE  EVENT 
(Second  Round  with  Safety  Mandate,  Grounded  Type) 


Expe  rt 

Event  ^ 

I 

2 

3 

4 

5 

6 

7 

E 

g 

0.  05 

0.  05 

0.  08 

0.  08 

0.  2 

0.  25 

Eo 

0.  1 

0.  000  l 

0.  005 

0.  0005 

0.  005 

0.  0001 

0.  25 

E1 

0.  001 

0.  00001 

0.  012 

0.  01 

0.  008 

0.  5 

0.  0005 

E2 

1.  0 

0.  0001 

1.0 

1.  0 

1.  0 

0.  001 

1.  0 

E3 

0.  01 

0.  005 

0.  008 

0.  008 

0.  008 

0.  05 

0.  05 

E4 

0.  04 

0.  002 

0.  0005 

0.  00003 

0.  001 

0.  002 

0.  02 

Eg 

0.  4 

0.  01* 

1.0 

1.  0 

, 

0.  7 

! 

0.  001 

0.  25 

E6 

0.  4 

0.  1 

m 

Ol 

0.  65 

0.  25 

E7 

0.  001 

0.  0001 

0.  00001 

0.  000001 

0.  00001 

0.  001 

0.  02 

E8 

0.  001 

0.  0001 

0.  00005 

0.  00001 

0.  00003 

0.  001 

0.  01 

TABLE  C-6 

ESTIMATED  PROBABILITY  OF  THE  EVENT 
(Second  Round  with  Safety  Mandate,  Doubly  Insulated  Type) 


^'"'"v^Expe  rt 

Event 

■ 

2 

3 

4 

5 

6 

7 

E* 

8 

0.  05 

0.  05 

0.  08 

0.  08 

0.  2 

0.  5 

0.  25 

Eb 

0.  1 

0.  0001 

0.  005 

0.  0005 

0.  005 

0.  0001 

0.  25 

E1 

0.  00001 

0.  012 

0.  01 

0.  008 

0.  3 

0.  0001 

E2 

1.  0 

0.  0001 

1.  0 

1.  0 

1.  0 

0.  001 

1.  0 

E3 

0.  005 

0.  005 

0.  008 

0.  008 

0.  008 

0.  05 

0.  05 

E4 

0.  05 

0.  002 

0.  0005 

0.  00003 

0.  001 

0.  002 

0.  001 

E5 

0.  1 

0.  01 

1.  0 

1.  0 

0.7 

0.  001 

0.  15 

E,6 

0.  001 

0.  00001 

0.  01 

0.  01 

0.  008 

0.  0005 

0.  01 

EV 

0.  0001 

0.  000001 

0.  00001 

0.  000001 

0.  00001 

0.  00002 

0.  01 

E 8 

0.  0001 

0.  00005 

0.  00001 

0.  00003 

0.  0015 

0.  001 
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TABLE  C-7 

SUMMARY  FOR  TABLE  C-5 
(Second  Round  with  Safety  Mandate) 


Event 

i 

Lowe  st 
Estimate 

Highest 

Estimate 

Median 

E 

g 

0.  05 

0.  5 

0.  08 

E0 

0.  0001 

0.  25 

0.  005 

E1 

0. 00001 

0.  5 

0.  008 

E2 

0.  0001 

1.  0 

1.  0 

E3 

0.  005 

0.  05 

0.  008 

E4 

0.  00003 

0.  04 

0.  002 

E5 

0.  001 

1.  o 

0.  4 

E6 

0.  1 

0.  7 

0.  5 

E7 

0.  000001 

0.  02 

0.  0001 

w 

oo 

0.  00001 

0.  01 

0.  0001 
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TABLE  C-8 

SUMMARY  FOR  TABLE  C-6 
(Second  Round  with  Safety  Mandate) 


Event 

Lowe  st 
Estimate 

Highe  st 
Estimate 

Median 

E' 

g 

0.  05 

0.  5 

0.  08 

E' 

Lo 

0.  0001 

0.  25 

0.  005 

E'l 

0. 00001 

0.  3 

0.  008 

E2 

0.  0001 

1.  0 

1.  0 

E3 

0.  005 

0.  05 

0.  008 

E4 

0. 00003 

0.  05 

0.  002 

E5 

0.  001 

1.  0 

0.  15 

E 6 

0. 00001 

0.  01 

0.  008 

E7 

0. 000001 

0.  01 

0.  00001 

Ek 

0.  00001 

0.  001 

0.  0001 

C-8 


TABLE  C-9 

(Second  Round  Without  Safety  Mandate) 


Eve  nt 

Lowe  st 
Estimate 

Highe  st 
Estimate 

Median 

E 

g 

0.  05 

0.  5 

0.  5 

E0 

0.  0001 

0.  25 

0.  1 

E1 

0.  00001 

0.  05 

0.  01 

E2 

0. 0001 

1.  o 

1.  0 

E3 

0.  005 

0.  05 

0.  01 

E4 

0.  002 

0.  05 

0.  05 

E5 

0.  8 

1.  o 

1.  o 

E6 

0.  1 

0.  4 

0.  25 

E7 

0.  0001 

0.  005 

0.  01 

E8 

0.  0001 

0.  05 

0.  001 
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TABLE  C-10 

(Second  Round  Without  Safety  Mandate) 


Event 

Lowest 

Estimate 

Highe  st 
Estimate 

Median 

E' 

g 

0.  05 

0.  5 

0.  5 

E'o 

0.  0001 

0.  25 

0.  1 

Ei 

0.  00001 

0.  01 

0.  01 

E2 

0.  0001 

1.  0 

1.  0 

E3 

0.  005 

0.  05 

0.  005 

E4 

0.  002 

0.  05 

0.  01 

E5 

0.  4 

1.  0 

1.  0 

,E6 

0. 00001 

0.  01 

0.  001 

E7 

0. 000001 

0.  01 

0.  0001 

E' 

^8 

0.  0001 

0.  01 

0.  0001 

OPTIMIST/PESSIMIST  CLASSIFICATION  OF  EXPERTS 


C-10 


-12 


C- 12 


TABLE  C - 1 3 

SUMMARY  STATISTICS  OF  OPTIMIST/PESSIMIST  SEPARATED  DATA 


Event 

Geometric 

Mean 

High 

Median 

Low 

Median 

E 

g 

0.  07  07 

0«  5 

0.  005 

E0 

0.  00937 

0.  175 

0.  00019 

E1 

0.  007  07 

0.  1 

0.  00028 

Ez 

0.  0316 

1.  o 

0.  0055 

E3 

0. 00837 

0.  05 

0.  0004 

E4 

0.  00265 

0.  05 

0.  000125 

E5 

0.  0316 

1.  o 

0.  006 

E6 

0. 0335 

0.  425 

0.  0035 

E7 

0. 000447 

0.  275 

0.  0000115 

E8 

0.  000316 

0.  01 

0.  000003 

C- 1 3 

TABLE  C-14 


SUMMARY  STATISTICS  OF  OPTIMIST/ PESSIMIST  SEPARATED  DATA 


C - 1 4 


TABLE  C - 1 5 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E 

g 

log  Eg 

Experts 

Distance  Between 
log  Estimate 

0.005 

-2.3010 

6,  9,  10 

> 1.301 

0.1 

-1.0 

12 

> 0.4771 

0.3 

-0.5229 

11 

> 0.2219 

0.5 

-0.3010 

1,  2,  4,  5 

> 0.1761 

0.75 

-0.1249 

3 

> 0.0791 

0.9 

-0.0458 

8 

> 0.0458 

1.0 

0 

7 

D = Average  Distance  = 2.3010  = 0.1917  0.192 

12 

3D  = 0.576 

Cluster  1 : 6,  9,  10 

Cluster  2 - 1,  2,  3,  4,  5,  7,  8,  11,  12 


/ 


C - 1 5 


TABLE  C-16 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


Eo 

log  Eq 

Expert 

Distance 

0.0001 

-4.0 

5,  9 

7 0.4472 

0.00028 

-3.5528 

6 

> 1.206 

0.0045 

-2.3468 

10 

> 0.3468 

0.01 

-2.0 

4 

> 0.3522 

0.0225 

-1.6478 

11 

> 0.6478 

0.1 

-1.0 

2,  3 

> 0.3979 

0.25 

-0.6021 

1 

> 0.4150 

0.66 

-0.1871 

7 

> 0.0902 

0.8 

-0.0969 

8 

>0.0404 

0.876 

-0.0565 

12 

>0.0404 

Average  distance  = 3.9435  = 0.3286  0.329 

12 

3D  = 0.987 

Cluster  1 - 5,  6,  9 

Cluster  2s  1,2,  3,  4,  7,  8,  10,  11,  12 


/ 


C-16 


TABLE  C-17 


AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E1 

log  Ej 

Experts 

Distance 

0.0001 

-4.0 

9,  10 

7 0.4472 

0.00028 

-3.5528 

11 

7 0.5528 

0.001 

-3.00 

3,  4,  12 

7 1.0 

0.01 

-2.0 

2 

7 0.699 

0.05 

-1.3010 

1 

> 0.3010 

0.1 

-1.0 

7 

7 0.3010 

0.2 

-0.6990 

8 

> 0.301 

0.5 

-0.3010 

5 

> 0.398 

0 

6 

not 

considered 

D = Distance  = 3.699  = 0.3083  ^ 0.308 

12 

3D  - 0.924 

Cluster  1 - 3,  4,  9,  10,  11,  12 
Cluster  2 = 1,  2,  5,  7,  8 


/ 


CI- 


TABLE C-18 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E2 

log  E 

0.001 

-3.0 

0.005 

-2.3010 

0.0055 

-2.2596 

0.007 

-2.1549 

0.01 

-2.0 

0.2 

-0.6990 

0.8 

-0.0969 

0.9 

-0.0458 

1.0 

0 

2 


5 

9 

10 

6 

11 

12 
4 
3 
1, 


Experts 


> 

> 

> 

> 


Distance 


0.699 

0.0414 

0.1047 

0.1549 


> 1.301 


2,  7,  8 


> 0.6021 

> 0.05T1 

> 0.0458 
>0.0458 


D = Distance  = 3.0  = 0.25 
12 

3D  = 0.75 

Cluster  1 = 5,  9,  10 

Cluster  2 * 6,  11 

Cluster  3 ■ 1,  2,  3,  4,  7,  8,  12 


/ 


■M 


C - 1 8 


TABLE  C-19 


AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E3 

log  E3 

Experts 

Distance 

0.0001 

-4.0 

9,  10 

> 0.8451 

0.0007 

-3.1549 

6 

? 0.047 

0.0078 

-2.1079 

12 

} 0.1079 

0.01 

-2.0 

3,  4 

> 0.0969 

0.0125 

-1.9031 

11 

7 0.6021 

0.05 

-1.3010 

2,  5 

> 0.3010 

0.1 

-1.0 

1,  7 

7 0.8451 

0.7 

-0.1549 

8 

D = 3.8451  - 0.3304  ^ 0.33 

12 

3D  =*  0.99 

Cluster  1 ■ 1,  2,  3,  4,  5,  6,  7,  8,  9,  10,  11,  12 


/ 


C-19 


TABLE  C-20 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E4 

log  E4 

Experts 

Distance 

0.00007 

-4.1549 

6 

.>  0.1549 

0.0001 

-4.0 

9 

> 0.1761 

0.00015 

-3.8239 

10 

> 1.1249 

0.002 

-2.699 

5 

> 0.398 

0.005 

-2.3010 

3 

> 0.301 

0.01 

-2.0 

11,  12 

> 0.699 

0.05 

-1.3010 

00 

r— 1 

> 0.301 

0.1 

-1.0 

2 

D = Average  Distance  = 3 . 1549  = 0.2629  — 0.263 

12 

3D  = 0.789 

Cluster  1 » 6,  9,  10 

Cluster  2 ’ 1,  2,  3,  4,  5,  7,  8,  11,  12 


> 


I 


C-20 

TABLE  C-21 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E5 

log  E5 

Experts 

Distance 

0.001 

-3.0 

5 

70.699 

0.005 

-2.3010 

9 

>0.0792 

0.006 

-2.2218 

10 

>0.0669 

0.007 

-2.1549 

6 

>0.4973 

0.022 

-1.6576 

12 

> 1.5027 

0.7 

-0.1549 

2 

>0.058 

0.8 

-0.0969 

3 

>0.0969 

1.0 

0 

1,  4,  7,  8,  11 

>0.0969 

D = Average  Distance  = 3__  = 0.25 

12 

3D  - 0.75 

Cluster  1 s 5,  6,  9,  10,  12 
Cluster  2 5 1,  2,  3,  4,  7,  8,  11 


/ 


♦ 


C-21 


TABLE  C -22 


AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E6 

log  E6 

Experts 

Distance 

0.0015 

-2.8239 

10 

> 0.3680 

0.0035 

-2.4559 

6 

> 0.1549 

0.005 

-2.3010 

9 

> 1.6020 

0.2 

-0.6990 

2,  3,  4, 

/ 0.2431 

0.35 

-0.4559 

1 

>0.0843 

0.425 

-0.3716 

11 

>0.1498 

0.6 

-0.2218 

7,  12 

> 0.0347 

0.65 

-0.1871 

5 

/0.0622 

0.75 

-0.1249 

8 

D 3 Average  Distance  = 2.699  = 0.225 

12 


3D  = 0.675 

Cluster  1=6,  9,  10 

Cluster  2 3 1,  2,  3,  4,  5,  7,  8,  11,  12 


? 


C-22 

TABLE  C-23 


AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


C-2  3 


TABLE  C-24 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E8 

lQg  Eg 

0.000002 

-5.6990 

10 

0.000003 

-5.5229 

6 

0.0001 

-4.0 

9 

0.00035 

-3.4559 

12 

0.001 

-3.0 

2. 

0.002 

-2.6990 

11 

0.01 

-2.0 

3 

0.02 

-1.6990 

7 

0.05 

-1.3010 

1, 

Experts 


Distance 


5 


> 0. 1761 

> 1.5229 
>0.5441 
>0.4559 
>0.3010 
>0.6990 

> 0.3010 

> 0.3980 


4,  8 


D » 4.398  = 0.3665  - 0.366 

12 

3D  - 1.098  1.1 


Cluster  1*6,  10 

Cluster  2 a l,  2,  3,  4,  5,  7,  8,  9,  11,  12 


C -24 


| 


| 

TABLE  C-25  | 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS  ] 


E' 

g 

log  E 

0.00005 

-4.3010 

0.0005 

-3.3010 

0.002 

-2.6990 

0.1 

-1.0 

0.3 

-0.5229 

0.5 

-0.3010 

0.75 

-0.1249 

0.  9 

-0.0458 

D = Average  distance  = 4.2552 

II 


3D  = 1.161 


Experts 

Distance 

. 

6 

> 1.0 

8 

> 0.602 

9 

> 1.699 

11 

7 0.4771 

10 

> 0.2219 

1,  2,  4,  5 

> 0.1761 

3 

> 0.1761 

7 


0.3868  ^ 0.387 


Cluster  1 m 6,  8,  9 

Cluster  2 5 1,  2,  3,  4,  5,  7,  10,  11 


C-25 


TABLE  C -26 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E * 

11  0 

l°g  E ' 0 

Experts 

0.00001 

-5.0 

8 

> 

0.00003 

-4.5229 

6 

> 

0.0001 

-4.0 

5 

7 

0.0018 

-2.7447 

9 

> 

0.01 

-2.0 

4 

> 

0.0225 

-1.6478 

10 

> 

0.1 

-1.0 

2,  3 

> 

0.25 

0.6021 

1 

> 

0.8 

0.0969 

7 

0.878 

0.0565 

11 

Distance 


0.4771 

0.5229 

1.2553 

0.7447 

0.3522 

0.6478 

0.3979 

0.5052 

0.0404 


D =*  4.9435  = 0.4494  C:  0.45 
11 

3D  =*  1.35 


Cluster  1 » 1,  2,  3,  4,  5,  6,  7,  8,  9,  10,  11 


t 


C- 26 


TABLE  C -27 


AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


F f 

1 

log  E ' ^ 

Experts 

Distance 

0.00001 

-5.0 

8 

> 0.6021 

0.00004 

-4.3979 

9 

>0.8451 

0.00028 

-3.5528 

10 

> 0.5528 

0.001 

-3.0 

4,  11 

> 1.0 

0.01 

-2.0 

1,  2 

> 0.301 

0.02 

-1.6990 

7 

> 1.1761 

0.3 

0.5229 

5 

0 

3,  6 

Not 

Considered 

D * 4.4771  = 0.4975  Cr  0.5 
9 

3D  » 1.5 

Cluster  1 = 1,  2,  4,  5,  7,  8,  9,  10,  11 


C -27 

TABLE  C-28 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


e'2 

log  E'2 

Experts 

Distance 

0.0005 

-3.3010 

8 

> 0.1461 

0.0007 

-3.1549 

6 

> 0.1549 

0.001 

-3.0 

5 

> 0.3424 

0.0022 

-2.6576 

9 

> 0.6576 

0.01 

-2.0 

10 

> 1.301 

0.2 

0.6990 

11 

> 0.6021 

0.8 

0.0969 

4 

> 0.0511 

0.9 

0.0458 

3 

> 0.0458 

1.0 

0 

1,  2,  7 

D = 3.301  = 0.3001  0.3 

11 

3D  =■  0.9 

Cluster  1 5 5,  6,  8,  9,  10 
Cluster  2 a 1,  2,  3,  4,  7,  11 


C-28 

TABLE  C-29 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E'3 

log  E'3 

Experts 

Distance 

0.00004 

-4.3979 

9 

> 0.3979 

0.0001 

-4.0 

8 

->  1.8921 

0.0078 

-2.1079 

11 

> 0.1079 

0.01 

-2.0 

3,  4 

> 0.0969 

0.0125 

-1.9031 

10 

? 0.2041 

0.02 

-1.6990 

2 

> 0.3980 

0.05 

-1.3010 

5 

> 0.3010 

0.1 

-1.0 

1 

> 0.8451 

0.7 

-0.1549 

7 

0 

6 

Not 

Cons idered 

D = 4.2430  = 0.4243  0.424 

10 

3D  = 1.272 
Cluster  1 s 8,9 

Cluster  2 * 1,  2,  3,  4,  5,  7,  10,  11 


I 


C-29 


TABLE  C - 30 


AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E'4 

iog  E'4 

Experts 

Distance 

0.00004 

-4.3979 

9 

> 0.3979 

0.0001 

-4.0 

8 

> 1.301 

0.002 

-2.6990 

5 

7 0.398 

0.005 

-2.3010 

3 

> 0.301 

0.01 

-2.0 

1,  10,  11 

7 0.301 

0.02 

-1.699 

7 

•>  0.398 

0.05 

-1.301 

4 

? 0.301 

0.1 

-1.0 

2 

7 0.301 

0 

6 

Not 

Considered 

D = 3.3979  = 0.3398  Gs  0.34 

10 

3D  = 1.02 
Cluster  1 ~ 8,  9 

Cluster  2 5 1,  2,  3,  4,  5,  7,  10,  11 


C-  30 


TABLE  C-31 


AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


£,5 

Log  E's 

Experts 

Distance 

0.0005 

-3.3010 

8 

> 0.1461 

0.0007 

-3.1549 

6 

7 0.1549 

0.001 

-3.0 

5 

? 0.3424 

0.0022 

-2.6576 

9 

>1.00 

0.022 

-1.6576 

ii 

> 1.5027 

0.7 

0.1549 

2 

>0.058 

0.8 

0.0969 

3 

0.0969 

1.0 

0 

1,  4,  7,  10 

D = 3.301  = 0.3001  0.3 

11 

3D  = 0.9 

Cluster  1 5 5,  6,  8,  9 

Cluster  2 = 11 

Cluster  3 3 1,  2,  3,  4,  7,  10 


C-3 


TABLE  C-32 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


D = 0.5221  Sr  0.522 
3D  = 1.566 
Cluster  1=9 

Cluster  2 = 1,  2,  3,  4,  5,  7,  10,  11 


C-  33 

TABLE  C- 34 

AVERAGE  DISTANCE  CLUSTERING  COMPUTATIONS 


E 1 
* 8 

log  e'8 

Experts 

Distance 

0.000002 

-5.6990 

9 

> 0.699 

0.00001 

-5.0 

8 

> 1.5441 

0.00035 

-3.4559 

11 

> 0.4559 

0.001 

-3.0 

2 

> 0.1761 

0.0015 

-2.8239 

5 

? 0.1249 

0.002 

-2.6990 

10 

> 0.398 

0.005 

-2.3010 

3 

> 0.301 

0.01 

-2.0 

1 

> 0.301 

0.02 

-1.6990 

7 

> 0.398 

0.05 

-1.3010 

4 

0 

6 

Not 

Considered 

D = 4.398  = 0.4398  - 0.44 

10 

3D  = 1.32 
Cluster  1 5 8,  9 

Cluster  2 5 1,  2,  3,  4,  5,  7,  10,  11 


